Skip to content

Pass 1: single-measure filtered aggregates - #608

Draft
zzylol wants to merge 3 commits into
stack/509-y2-filtered-aggregatesfrom
stack/509-y2b-filtered-pass1
Draft

zzylol wants to merge 3 commits into
stack/509-y2-filtered-aggregatesfrom
stack/509-y2b-filtered-pass1

Conversation

@zzylol

@zzylol zzylol commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Stack: Wave 3 chain: #605 → #607 → #608 → #609 → #610

Rebased on main d4869a7 (DF 54).

Stack: … → #605 → #607 → this PR

Why

Part of #509 and #580 (J, filtered aggregates); Q47 (a). Pass 1 listed alternatives for a filtered single-measure aggregate, but realize rejected every non-pass-through choice ("filtered or HAVING aggregate"). So these alternatives failed in Stage 1, and the query could only run as written, which #607 makes possible. This covers SQL FILTER (WHERE …) and COUNT(y) over a nullable y, which is lowered as a count filtered by y IS NOT NULL.

What

SELECT g, COUNT(*) FILTER (WHERE x > 0) AS c FROM events GROUP BY g at ε = 0.1, δ = 0.01:

alternatives that compose
Before this PR pass-through only. Count acc, CMS, CountSketch, UnivMon and HydraCms fail: unsupported local realization: filtered or HAVING aggregate
After this PR all 6. Each SummaryAgg carries filter: x > 0. An alternative binds in the executor exactly when its unfiltered counterpart does. The Count accumulator and HydraCms execute to a 2, b 0, c 1, matching the exact plan.

For a filtered SUM and a filtered approx_percentile_cont, the exact Sum accumulator, KLL and DDSketch execute to the exact answer. They read NULL for the group with no matching row.

Selection: the Examples 1, 3a, 3b and 4a stage documents are byte-identical to #607's, and the committed Example 1 fixture is unchanged. Example 2 has no filtered measure (planner_layering_example2 asserts this), and its tests are unchanged. I did not tune costs.

How

  • Pass 1 (logical_candidates::realize): a single measure's filter becomes SummaryAgg.filter, which reads the same input rows. This applies to every family, Hydra, and tumbling panes, since each pane carries the filter. HAVING stays rejected. The SQL frontend lowers it to a Filter above the aggregate anyway. A whole-expression (absorbing) target is never filtered.

  • Coverage stays whole-source (Check declared summary coverage population against subtree filters #570: the declared population is trusted). A filtered state holds a subset of its source, so whole-source over-states its population. That is the conservative direction:

    • merges need proven disjointness, and a larger claim proves less;
    • CSE compares the operator, filter included, so a filtered state is never shared with an unfiltered one.

    A column = literal filter could be declared as that population, but nothing reads populations from Pass 1 yet. I did not do this.

  • Types: FinalizeExactAccumulator and a quantile SummaryEstimate over a filtered build are declared nullable, like the filtered aggregate each realizes. Without this, a filtered exact SUM read 0.0 for an empty group.

Tests

These failed or did not exist on #607:

  • filtered_aggregates::pass1_offers_filtered_count_alternatives: the six alternatives, the filter on each, bind parity with the unfiltered plan, and execution matching the exact plan.

  • filtered_aggregates::pass1_filtered_sum_and_quantile_alternatives_execute: SUM (exact acc) and percentile (KLL, DDSketch within 2%), with NULL for the empty group.

  • stage_pipeline_selection::sql_filtered_aggregates_dp_equals_exhaustive: a filtered grouped count × a filtered percentile, 18 combinations. The DP equals exhaustive, and none is rejected while being built.

  • summary_coverage_e2e::sql_worked_example: the docs: ASAP primitive schema and summary coverage (definition + selection) #573 §4.2.2 worked example, FILTER (WHERE latency < 100) over a renamed, filtered subquery. Its KLL state's selection is latency ∈ (−∞, 100), r ∈ {us}, and the definition keeps no filter on the SummaryAgg. The test reads the state from the Stage 1 candidates, since the selected plan for such a small query may build no summary.

filtered_aggregates::exact_filtered_sum_is_null_for_groups_without_matches from #607 now selects the filtered Sum accumulator, and it still reads NULL.

Not in this PR

  • Multi-measure split: an aggregate with several measures, filtered or not, stays a single pass-through target. Splitting it into single-measure targets, each with its own filter, is a follow-up.
  • HAVING on Aggregate stays rejected. The frontends never emit it; SQL HAVING is a post-aggregate Filter.
  • Gaps that predate this PR and that filters do not change: a per-group CMS/CountSketch for SQL COUNT(*) does not bind (keyed summary weight must be a finalized value column), and filtered ingestion-time precompute is unsupported (executor: filtered aggregates and summary builds #607).

Gates

Refs #509, #580, #605.

🤖 Generated with Claude Code

zzylol and others added 3 commits October 11, 2026 16:22
Pass 1 listed alternatives for a filtered single-measure aggregate (SQL
FILTER, or the IS NOT NULL a COUNT(nullable) implies) but every
non-pass-through choice failed to compose. The measure's filter now
becomes SummaryAgg.filter, so exact accumulators and sketches (Hydra
included) compose, build, and execute. Coverage stays whole-source.

A FinalizeExactAccumulator or quantile SummaryEstimate over a filtered
build is declared nullable, as the filtered aggregate it realizes is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pass 1 now builds a filtered SummaryAgg, so the #573 §4.2.2 worked example
runs whole: the FILTER range and the renamed value set move to the
selection, and the expression condition stays in the definition. The
quantile state is read from the Stage 1 logical candidates, which build it
whatever plan the cost model selects.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
@zzylol
zzylol force-pushed the stack/509-y2-filtered-aggregates branch from 6ba89f0 to 579b2d4 Compare October 11, 2026 17:11
@zzylol
zzylol force-pushed the stack/509-y2b-filtered-pass1 branch from 2faa0de to 0bbfc1b Compare October 11, 2026 17:11
zzylol added a commit that referenced this pull request Oct 13, 2026
… targets

Stage 1 composition realizes every #561 Pass 1 choice. A measure filter
becomes SummaryAgg.filter (the realization rule #608 carried, with its
filtered-output nullability in ASAPOp schemas); a key-only HAVING that
every consumer applies becomes a Filter below the state (SummaryAgg.filter
would keep the dropped groups); a value HAVING stays a Filter above the
output. avg divides its finalized sum and count accumulators, and the
measures of a multi-measure aggregate are realized one by one and joined
back on the group keys. ScalarExpr::map_columns renumbers a predicate's
columns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
zzylol added a commit that referenced this pull request Oct 13, 2026
… targets

Composition substitutes every #561 Pass 1 choice. A measure filter
becomes SummaryAgg.filter (the realization rule #608 carried, with its
filtered-output nullability in ASAPOp schemas); a key-only HAVING that
every consumer applies becomes a Filter below the state (SummaryAgg.filter
would keep the dropped groups); a value HAVING stays a Filter above the
output. avg divides its finalized sum and count accumulators, and the
measures of a multi-measure aggregate are substituted one by one and joined
back on the group keys. ScalarExpr::map_columns renumbers a predicate's
columns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW
zzylol added a commit that referenced this pull request Oct 13, 2026
…mputation candidates (#575)

* feat(planner): compose a whole-workload candidate from Pass 1 choices

Replays #575's composition on the #561 Pass 1 API: choice[i] indexes
targets[i].choices.candidates, several targets (measures) of one
aggregate are grouped, an exact accumulator becomes SummaryAgg →
FinalizeExactAccumulator and a summary SummaryAgg → SummaryEstimate.
Pass 1 summaries are unsized, so composition sizes each for its measure's
accuracy requirement. Summary coverage is derived from the subtree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* fix(ir): name a finalized exact accumulator after the aggregate it realizes

FinalizeExactAccumulator kept the state column's name (`state`), so a
consumer reading the sample value (e.g. sum by (job) over a finalized
per-series rate) found no `value` column at runtime.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(planner): rank Top-K summary items by the PromQL series identity column

Rows that carry the full series identity are closed; Pass 1 rejected them
and the runtime needs explicit item columns for keyed summaries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(devtools): stage_pipeline writes real frontend and logical ASAP DAG documents

Replays #575's devtools commits on the #561 Pass 1 API: the frontends section is the
frontends' workload DAG and logical_asap_optimization holds one composed whole-workload
candidate per choice of a per-computation candidate for every target, as flat DAGs,
capped by --max-candidates. Example 1 has three candidates (one exact
option per target), so the cap test cuts at two.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* feat(planner): compose measure filters, HAVING, avg and multi-measure targets

Composition substitutes every #561 Pass 1 choice. A measure filter
becomes SummaryAgg.filter (the realization rule #608 carried, with its
filtered-output nullability in ASAPOp schemas); a key-only HAVING that
every consumer applies becomes a Filter below the state (SummaryAgg.filter
would keep the dropped groups); a value HAVING stays a Filter above the
output. avg divides its finalized sum and count accumulators, and the
measures of a multi-measure aggregate are substituted one by one and joined
back on the group keys. ScalarExpr::map_columns renumbers a predicate's
columns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* feat(planner): give composed logical ASAP candidates theoretical guarantees

Every node of a composed candidate gets a guarantee: exact over exact
inputs, a summary estimate's sized family bound over exact input, and None
for anything computed from an approximate value, including a value HAVING
or any other predicate at any depth (#669 defines composition; plan selection's
accuracy model decides). The Example 1 fixture shows them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* fix(planner): keep summary candidates for combinations of approximate values

A comparison of approximate operands (quantile(0.99, …) > quantile(0.5, …))
now gets the same BinaryOp candidate over summary operands as arithmetic
does, so no combination loses its summary candidates. Automatic selection
still needs evidence about the candidate's output: a root guarantee that
meets the target, which composition (#669) supplies, or the deployment's
empirical output-level accuracy model (plan selection, #576). Certified inputs are
not enough, since a difference or ratio of them can be unbounded, so such
candidates are kept but not selected under the built-in models.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* docs(planner): composition of Pass 1 candidates in the module doc

The composition of per-computation candidates into whole-workload
candidates is Algorithm 1's SUBSTITUTE; its helpers are named so. The
module doc describes it and states that summaries are sized here per
measure because Pass 2 sharing is not implemented yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* fix(planner): no frequency bound on Top-K or signed Count-Min estimates

A heap sketch's bound is per item frequency, not Top-K membership, and a
Count-Min bound assumes non-negative weights. A composed Top-K estimate,
and a Count-Min estimate over weights not proven non-negative, now get no
guarantee; plan selection's accuracy model decides them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* fix(planner): rank Top-K over a closed PromQL child by its labels

topk(3, sum by (job) (rate(m[5m]))) listed heap candidates that composition
could not substitute: the closed child [job, value] has no series identity.
Its item is now the tuple of labels outside the partition, as in the legacy
current-series and counter rules. stage_pipeline records a candidate it
cannot compose with its error instead of aborting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* fix(planner): substituted outputs keep the aggregate's column names

An estimate or accumulator names its value after the operation
(quantile_0_99 became quantile, avg sum, a per-series count value count).
A single-measure substitution is now renamed to the aggregate's columns,
as join_measures already did; a renaming projection keeps its child's
guarantee. The logical_asap_optimization test compares column names too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* fix(planner): substitute subqueries inside measure filters and HAVING

A subquery in a substituted aggregate's measure filter or own HAVING now
reads its rewritten plan, and a summary state's guarantee requires every
input, its filter's subqueries included, to be exact.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* fix(ir): a filtered quantile aggregate is nullable

As its summary estimate is: no row passing the filter gives NULL. The
filtered quantile queries join the SQL candidate composition test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* fix(planner): compose SQL avg as a keyed join of sum and count

A PromQL BinaryOp matches Utf8 labels: it merges NULL and '' groups and
rejects other key types. For SQL rows the finalized sum and count are
joined on the group keys (NULL keys match) and divided as SQL, NULL where
the count is 0 (a filtered-empty group, an empty global input). PromQL
keeps the BinaryOp.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* fix(planner): count states read a column non-null on every counted row

The count state behind avg-as-sum/count and SQL count read the sample
value, which for SQL rows is `value` or the only numeric column: avg(c)
divided by count(value), count(*) counted a nullable column, and with
several numeric columns the composed DAG was invalid. The runtime skips
NULL updates, so:

- avg's count reads the averaged column (non-null, or the candidate is
  pruned);
- SQL count reads a column its filter proves non-null (count(col)), else
  a non-nullable column; with none it is Unsupported;
- an unresolvable state input is Unsupported, and a composed DAG is
  validated before it is returned.

PromQL counts still read the sample value.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

* test(planner): a subquery in an aggregate's own HAVING is substituted

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W7qG9aFyPij5uWsyAJCxDW

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant